texts: Why Uniformity of subpixel in image analysis ensures correctness

texts:
I don't know if its right place to ask this question.
I was working on particle tracking of series of pictures (.tiff) of colloids. And I was Using trackpy in python.
The code gives me the position of particles to subpixel accuracy.

There is another step involved is of checking the subpixel accuracy. In this method we check the uniformity in the decimal part of the position

a quick way to check for subpixel accuracy is to check that the decimal part of the x and/or y positions are evenly distributed. Trackpy provides a convenience plotting function for this called subpx_bias

Mask size is more or less the diameter of the particle.
I am not getting how does evenly distribution of decimal part ensures that we are on the right track?
And how a dip shows that we are wrong somewhere?
You can also refer to Eric weeks website. on this website, it is briefly mentioned that

One failure mode is if the length scale in the feature is made too small, then all the x and y coords get 'rounded off' to the nearest pixel value. The above command plots a histogram of the fractional part of the x-coords of the image. The physically distributed positions should be random -- giving a flat histogram. If the histogram has two peaks (near 0 and 1), set the size parameter in 'feature' a little bigger, determine a new masscut, and repeat until everything is happy.


texts: Where does the sum of squared errors function in neural networks come from?

texts:
Training a basic multilayer perceptron neural network boils down to minimizing some kind of error function. Often the sum of squared errors is chosen as a this error function, but where does this function come from?

I always thought this function was chosen because it makes sense intuitively. However, recently I learned that this is only partly true and there is more behind it.

Bishop wrote in one of his papers that the sum of squared errors function can be derived from the principle of maximum likelihood. Furthermore he wrote that the squared error therefore makes the assumption that the noise on the target value has a Gaussian distribution.

I am not sure what he means with that. How does the sum of squared errors relate to the maximum likelihood principle in the context of neural networks?


texts:
I am very new to Data Science, but I have an use case which I want to solve.

I want to build a data synchronization scheduler which keeps track of the amount of data sync after every scheduled triggers and auto-adjusts the next schedule.

For example :

Let us suppose I have 3 jobs to execute. Currently, we keep each of
them at 5 minutes interval (say) but this needs to be auto-scheduled.

So let at 10 AM Job 1 got executed and got 10 entries.
At 10 AM job 2 got executed and got 100 entries.
At 10 AM job 3 got executed and got 200 entries.

For such a scenario, job 1 got less stream of data than job 2 and job 3. The auto-scheduler in such a case will auto-adjust the interval and recommend to change the next execution at :

Job 1 - may be 10 min interval
Job 2 - may be 5 min interval
Job 3 - may be 2 min interval.

The scheduler will train itself based on time based historical data as well, for instance if stream of data is more at 10 AM for job 3, it might be less at 1 PM, when Job 1 might have more data. The scheduler would automatically adjust the time and make next schedule of 1 PM at lesser time interval for job 1 than job 3.

Can you suggest me any algorithm which I can follow to support this case ? Or even if you can help me how to proceed in ML, it would help me a lot.


texts:
I recently started working as a data scientist and I am starting a web scraping and NLP project using Python. The idea is to create a program that searches for public information on the company's clients. These information can come from various sources: annual reports, income statements, articles.... I will have to deal with two types of formats: HTML and PDFs. For now I will focus on retrieving the revenue of the company. After a month of research and tests, I realized a few things:
- NLP techniques are too slow to be used on annuals reports
The first step of the project will be the following:

Search for the annual report and scrape the HTML code: so far I managed to get all the google results and I'm using Beautifulsoup to get the HTML code. However I can't quite get the revenue of the company because each website has its own HTML structure. I first decided to focus on extracting tables (the goal is to find the company's income statement) but I realized that HTML tables are often used for layout (even if it's a bad practice). I can't rely on css selectors as I need to keep it as generic as possible. How can I achieve it?


texts: How to compensate for class imbalance in prediction model?

texts:
I'm trying to run a prediction model on a customers' data set to predict the likelihood that a new customer would be interested in buying product X, offered by a company that sells products X,Y and Z. E.g. would this guy, non-customer, of this age and salary, be interested in product X?

To train the model, I have a basin of 100K company customers, of which only 5K bought product X - the remaining 95K bought other products. Any prediction model guesses 'nobody will buy product X' accepting those ~5% false negatives.

How can I compensate for this skewness of the data? i.e. 95% vs. 5%? Thanks


texts: What is a 1D Convolutional Layer in Deep Learning?

texts:
I have a good general understanding of the role and mechanism of convolutional layers in Deep Learning for image processing in case of 2D or 3D implementations - they "simply" try to catch 2D patterns in images (in 3 channels in case of 3D).

But recently I bumped into 1D convolutional layers in the context of Natural Language Processing, which is a kind of surprise for me, because in my understanding the 2D convolution is especially used to catch 2D patterns that are impossible to reveal in 1D (vector) form of image pixels. What is the logic behind 1D convolution?


texts:
Can anyone help me identify what kind of Architecture is behind this Application? Is it a "simple" Classification Network? If so how are the heatmaps generated?

https://www.youtube.com/watch?v=c_giaOUzB_8

Thanks


texts: word2vec word embeddings creates very distant vectors, closest cosine similarity is still very far, only 0.7

texts:
I started using gensim's FastText to create word embeddings on a large corpus of a specialized domain (after finding that existing open source embeddings are not performing well on this domain), although I'm not using its character level n-grams, so it's basically just word2vec.
I'm testing the results by looking at some of the "most similar" words to key and the model seems to be working very well, except that the most similar words get at most a similarity score (using cosine similarity, gensim's FastText most_similar() function) of 0.7; versus google's or spacy's embeddings synonyms tend to have ~0.95+ similarity. I'm wondering why they don't in my case. This is not necessarily a problem, but it is not clear to me why this would happen; how come even very similar words, with very similar uses only get 0.7 in similarity? Again, the similar words themselves do make a lot of sense, so the training went well, but the score is kind of low.
My best guess is that maybe the embedding size I'm using is too big - 256. Can anyone suggest other explanations?
I trained it using CBOW, 15 epochs, window size=4, a few tens of millions of documents but only ~100000 words in the vocabulary that I'm interested in.


texts: Data preprocessing: Should we normalise images pixel-wise?

texts:
Let me present you with a toy example and a reasoning on image normalisation I had:

Suppose we have a CNN architecture to classify NxN grayscale images in two categories. Pixel values range from 0 (black) to 255 (white).

Class 0:
Images that contain a filled circle in the centre of the image. The pixel values within the circle are high, close to 255. Pixels outside the circle are low, close to 0. Below is an example of a class 0 picture.

Class 1:
Images that are empty, containing pixels close to 0. An example is below, basically a dark image.

Note: The images show basically binary values: white/black, however, assume that black values could be in the range of [0, 30] and white ones [230-255] with some random distribution.

Prior to feeding the network with the images, we pre-process and normalize the data. This basically means (i) centring the data and (ii) scaling the data. A typical and simple approach could be to subtract 127.5 to all pixels and scale by 255 so as to have pixel values ranging [-1, 1]. Alternatively, another approach consists of computing the "mean image" and the "dynamic-range image" and centre and scale the images pixel-wise. Let us focus on the latter.

Computing the "mean image" mainly means summing all images pixel-wise and taking the average pixel-wise. As a result, we obtain a NxN average image. This can sometimes shed some light on our task and help us understand the data we are working with. Likewise, the "dynamic range image" is obtained by computing the maximum and minimum values pixel-wise. The "dynamic range image" shows us which range of values each pixel can take.

Hypothesis 1

During training, the kernels of the different layers in the CNN will be trying to detect the circle. Since the task is rather simple, the kernels basically would have to learn that there is a strong pixel-value-transition from "empty region" to "circle region" when a circle is present.

Hypothesis 2

After normalising the images pixel-wise, we have that all pixels have the same range and same mean, namely [-1, 1] and 0, respectively.

Final Hypothesis

Consider a class 0 image (i.e. image has a circle). Before normalizing, pixel (i, j) has value 30 (dark), which happens to be the maximum that pixel (i, j) ever takes. Pixel (i, j+1) has a value 255 (white), also the maximum value that pixel (i, j+1) ever takes. So we have a strong transition that a kernel from first layers should be able to learn.

But, if we normalise the data with "mean image" and "dynamic range image"
isn't this strong transition going to be softened? In the particular, the transition would be from value 1 to value 1, since both were the maximums at their respective positions. From a clear and hard transition we have ended with a transitioned from similarly valued pixels.

Conclusion
Using this pixel-wise normalization we end up losing global pixel information, which might make the job of the kernels more difficult. I believe same conclusions might be drawn if "variance image" is used instead of "dynamic range image".

Question
Am I missing some point or is this normalisation no longer used?

Bonus
I also thought of Batch Normalisation, which sometimes is used at the first layer so that normalisation is done within the network. I thought that maybe the parameters "beta" and "gamma" used in BN might be able to fix this issue if it ever appears while training.


texts: Handling hierarchical category independent variables

texts:
I have data with huge categorical attributes.

For example, main_column, sub_column1, sub_column2 are 3 hierarchical attributes.
If if take dummy variable on these columns the column count is increased to 1000.

How to handle this kind of hierarchical attributes for a classification problem ?

Thanks !!


texts: Want suggestions on choosing open source embedded BI tool

texts:
I need some suggestions from all group members regarding the open source reporting/dash-boarding tool which fulfills below specific requirements apart from some basic BI functionalities:
1. Can be embedded inside a web
2. Interactive (filters and one chart selection affecting other charts)
3. Able to give excel downloadable option for plotted data
4. Role based user access

You can suggest tools supporting all requirements or partial also.


texts: Stabilize Neural network prediction for class probability

texts:
I could not carry my question from stackoverflow
I ve been trying to fit a neural network for binary setting using library(keras) and I am interested in class probability (instead of 0/1, probability of the event)
I ve 5.018 times more negative than positive class. I added the code I have been using. I cannot stabilize the predictions. I understand that noise and everything.
But I need to put some constraints to get close estimates each time. I am out of ides. Is there anything else I can use to stabilize predictions?
I cannot share the data therefore here is summary of predictions at train data level and I plotted validations/train.
 first run               Second run
 Min.   :0.001843       Min.   :0.0004508
 1st Qu.:0.012272       1st Qu.:0.0156236
 Median :0.042264       Median :0.0459510
 Mean   :0.142551       Mean   :0.1400624
 3rd Qu.:0.195536       3rd Qu.:0.1937293
 Max.   :0.919892       Max.   :0.9882065

validation plot for first run  and validation plot for second run
l2_model <-
  keras_model_sequential() %>%
  layer_dense(units = 512, activation = "relu", input_shape =  ncol(XX_train1),
              kernel_regularizer = regularizer_l2(0.001)) %>%
  layer_batch_normalization()%>%
  layer_dense(units = 256, activation = "relu",
              kernel_regularizer = regularizer_l2(0.001)) %>%
  layer_batch_normalization()%>%
  layer_dense(units = 1, activation = "sigmoid",
              bias_initializer = initializer_constant(log(5.0189)))

l2_model %>% compile(
  optimizer="Adam",
  loss = "binary_crossentropy",
  metrics =  c('accuracy')
)

summary(l2_model)

l2_history <- l2_model %>% fit(
  x                = as.matrix(XX_train1),
  y                = YY_train1,
  epochs = 30,
  batch_size = 1000,
  validation_data = list(XX_test, YY_test[,2]),
  verbose = 2,
  callbacks = list(
    callback_early_stopping(patience = 2) )
 #   ,callback_reduce_lr_on_plateau()  )
)

# Predicted Class Probability
yhat_keras_prob_vec  <- predict_proba(object = l2_model, x = as.matrix(XX_train1)) %>%
  as.matrix()

summary(yhat_keras_prob_vec)

I have look at some questions to get ideas but did not work
neural-network-loss-function-for-predicted-probability
Understanding neural network probability


texts: What is a good way to handle nominal spatial data with a changing number of categories to use in prediction model?

texts: For a project I'm going to be working with spatial data with a nominal attribute (land use). Every year the number of categories for this attribute changes because categories split or merge. I do have access to a chart that shows me how the categories are transformed into each other from one year to another. For the same spatial extent, I also have data for a bunch of other variables. I want to use these as explanatory variables for the nominal variable mentioned before. My question is now: How do I build a prediction model when the number of possible values for the reponse variable changes every year?


texts: multiple linear regression with 5 records and 25 features

texts:
X1     X2    X[...]    X25    Y

Q1_2019    23     65    18        32     1,6
Q2_2019    87     32    23        46     1,2
Q3_2019    34     15    63        78     3,2
Q4_2019    85     45    43        65     3,9
Q1_2020    85     43    78        35     1,1
Q2_2020    37     78    54        78     1,5

I have a very expensive dataset which shows aggregated survey data.
These are probably means. I am trying to get the individual data but
at the moment that is all I have.
The shape of data frame is 5x26
Y data so far is collected data calculated at the end of each quarter
via other means The survey is done at the beginning of the quarter.
Y is my dependent variable and I would like to derive a polynom to
predict the exact number based on future X data or at least the
probable trend it will be going in the next quarter once new survey
data is available. Up, down, stable would be enough
I have done correlation analysis (all vs all) and there are strong
pairwise    correlation between several X and Y

Questions

Y comes as a one digit before the comma and one digit after the comma. Since all other values are 2 digits before the comma I would like to multiply it with 10 to transform it into 2 digits before the comma.Is that ok from math/data science perspective?
5 records is not much but there are a lot of features. I would like to do multiple linear regression. Do you think this feasible with this data set? What would be objections and risks doing that?
Would upsampling the dataset help me with anything here? Or could I just work with the five records?
With the strange shape of the dataset especially the low number of records do you think that sufficient precision can be reached?
How could I calculated the maximum possible precision/discriminative power possible with this dataset? (I am looking for strong arguments why they should give me access to the complete dataset)


texts: How is bayesian risk computed to prune decision trees?

texts:
I've been trying to follow this paper on Bayesian Risk Pruning. I'm not very familiar with this type of pruning, but I'm wondering a few things:
(1) The paper describes risk-rates to be defined per example. We have $R_k(a_i|x)=\sum\limits_{j=1,j \neq i}^{T_c} L_k(a_i|C_j)p_k(C_j|x)$. $L_k(a_i|C_j)$ is defined to be the loss of an example being predicted in class $C_j$ when the true class is $C_i$. $p_k(C_j|x)$ is the estimated probability of an example belonging to $C_j$.

Above is the decision tree that was produced from a C4.5 algorithm. Pruning occurs from left to right, bottom-up. My main question: How are the risk-rates found in the decision tree such as for Node 3?
(2) There's also conflicting statements here:

The first image states that if parent risk-rate exceeds the total risk-rate of the leaves, then the parent is pruned to a leaf. However, the second claims that pruning occurs if the leaf risk-rate exceeds the parent. To confirm, if the risk-rate of the parent is less than the risk-rate of the leaves under the subtree of the parent, then I would set the parent to be a leaf?
(3) From (1), loss would be 0-1 in the binary case. What could be a reasonable loss for multi-class output?
(4) From (1), would estimated probability of $C_j$ be the proportion of $C_j$ in the partitioned output class at a node? For instance, at node 3, we're looking at output = [No].
(5) From (1), would risk-rate be over all training examples?


texts: Does discretization of continuous features also lose information about distance?

texts:
During discretization, it "squashes" nearby values into one bin, losing a little bit of information along the way.

But doesn't it also lose information about distances of features? For example, if we have height as continuous feature, we can e. g. create bins very small, small, medium, large and very large. Isn't it problem, that once we have these categories, we lose information that very small and small are closer than small and very large?

This would be even worse for predicted feature - I would assume that regression that tries to predict height number would be much more successfull than classifier that tries to predict category of height discretized, because cost function of regression can account for a distance from correct height, but cost function of classifier could only answer "correct" or "not correct".

Yet I haven't found any mention about this when I've searched about discretization.

Are my assumptions incorrect?


texts: inner workings of Mobile-net resolution multiplier - what does it do?

texts:
i have a question concerning the way Mobile Net's resolution parameter works. From the article itself and from the blog posts on the topic (1, 2) I wasn't able to find an answer to my question. It is said that “the resolution multiplier changes the input dimensions of the image, reducing the internal representation at every layer”. Does that mean it (the parameter):

A) breaks down each image into a, for example, respective 224x224 bits of pixels sub-images, over which the network runs its convolution?

OR

B) takes one single 224x224 slice of pixels from, for example, the middle of the image, over which the network runs its convolutions?

OR

C) Squishes the original image into a, for example 224x224 pixel representation of itself, over which the network runs its convolutions?


texts: Customer Segmentation: Should I use a variable, representing a product, that is unpopular in the dataset for K-Means Clustering?

texts:
I am working with a data set that, besides customer age and income, tells the balance a customer has in different type of bank accounts: Checking, Shares, Investment, Savings, Deposit, Mortgage, Loan, and Certificates. For accounts other than Checking, 0 represents that the account does not exist for the customers. There are 9800 customer observations with roughly 6000 checking accounts and 4000 savings accounts. For the others, the observations are less than 300.

I have to use K-Means Clustering analysis for the segmentation with the objective to understand how customers use savings and investment offerings and I am using the Elbow Method to predict the number of clusters. I am confused whether to use a variable like Investment with just 250 observations with another like Savings that has 4000. If I do use such variables, then these are heavily positively skewed and I'm not sure if K-Means handles that well. Can someone advise whether to include such variables or not?


texts: How to resize MNIST images to fit AlexNet model

texts:
I am using the keras API to load in the MNIST dataset. My problem is I need to use AlexNet as my algorithm. Understanding the AlexNet model, I require to start with 277x277 images but the MINST dataset has 28x28. How can I reshape the numpy array so that each image is 227x277 to then use the full AlexNet model?
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

This is how I load my data in. Could someone show me the solution to change the initial images to match the AlexNet model?


texts: Tips for improving multitask learning based on multiple outputs

texts:
I'm currently trying to use multi-task learning based on a multi-output model that both allows to get an output for classification and regression. However, at the moment it's staying at around 20% accuracy. I tried out multiple things including choosing multiple loss functions, weighting loss with loss_weights in keras. Further I tried to adapt my Adam optimizer to for different beta_1 and beta_2 values.
Since I read that it's better to share more in case of overfitting I tried out the following architecture:
def multitask_model(optimizer='rmsprop', init='glorot_uniform'):
    ampl_signal = Input(shape=(X_train.shape[1:]))
    phase_signal = Input(shape=(X_train_phase.shape[1:]))
    temp = Input(shape=(1,))
    moist = Input(shape=(1,))
    weight = Input(shape=(1,))

    reg = Concatenate()([ampl_signal, phase_signal])
    clf = Concatenate()([ampl_signal, phase_signal])

    concat_signal = Concatenate()([reg,clf])

    x = Permute(dims=(1,2))(concat_signal)
        #x = BatchNormalization()(x)

    x = Conv1D(128, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(128, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)

    x = Conv1D(128, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(128, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)

    x = Conv1D(64, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(64, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)

    x = Conv1D(64, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(64, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)

    x = Conv1D(32, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(32, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)

    x = Conv1D(32, 3, activation='LeakyReLU', kernel_initializer='he_uniform')(x) #, input_shape=(None, 3750, n_features)
    x = Conv1D(32, 3, activation='LeakyReLU',  kernel_initializer='he_uniform')(x)
    x = MaxPooling1D(3)(x)
    x = Dropout(0.3)(x)

    x = GlobalAveragePooling1D()(x)
    concatenated_features = Concatenate()([x,temp,moist, weight])#inputD
    x = Permute(dims=(1,))(concatenated_features)
activation='elu',kernel_initializer='glorot_normal')(x)

    classifier=Dense(1,activation='softmax')(x)
    regression=Dense(1,activation='linear')(x)

    mdl_classifier = Model(inputs=[ampl_signal, phase_signal, moist,temp, weight], outputs=classifier)
    mdl_regression = Model(inputs=[ampl_signal, phase_signal, moist,temp, weight], outputs=regression)
    mdl = Model(inputs=[mdl_classifier.inputs], outputs=[classifier,regression])

    mdl.compile(loss=['binary_crossentropy', 'logcosh'],
              optimizer = Adam(lr=0.0005,clipnorm = 1.,beta_1=0.9, beta_2=0.999, epsilon=0.00000001, amsgrad=True),

    return mdl #mdl_classifier, mdl_regression

Do you have any other idea what else I could do?
Best regards,


texts: What is the best way to deal with imbalanced data for XGBoost?

texts:
There are a lot of way to deal with class-imbalanced data like undersampling, oversampling, changing cost function etc.

https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/

Here is the post talking about them all.

I am working with an imbalanced multiclass classification problem and trying to solve it using XGBoost algorithm. I wanted to understand which method works best here. Since XGBoost already has a parameter called weights (which gives weight to each train record), would it be wise to directly use it instead of undersampling, oversampling, writing a cost function etc.?


texts: Do I need to use Bayes to combine a sample's class probability with the performance of the overall model?

texts:
For a classification algorithm that gives the predicted probabilities for each class (ie random forest in sklearn), by default the classes are separated with a score of 0.5. After evaluating the metrics, it looks like there would be a better precision/recall balance for my needs with a higher score.

If I use a split of 0.6 score, which results in a 0.8 precision and 0.9 recall, how do I communicate the overall confidence of future sample scores?

For example, if a sample returns a predicted probability of 1 vs one that returns a 0.75. Based on the threshold chosen, both are going to be considered as part of the positive class. However, it's estimated that only 80% of the time that will be true. On top of that, the 1 and 0.75 scores seem to indicate another level of confidence for the model.

This feels like Bayes should be used here to summarize the scores for interpretation. But, I am not sure. So, given the model has an estimated performance based on a score threshold, and given some scores, how do you give an overall percentage of the samples actually being in the positive class? And how would one communicate this to non-technical people?


texts: How to Access a Text File from IBM DSX Scala-Spark Notebook

texts:
I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to me that the general configuration of the platform is focused on CSV format; so is most of the help material. In fact, I figured a way around when I changed the extension of my .txt file to .csv. Through this post and the answer followed, I am trying to share my experience with others.

This post is useful, if:

You have created a IBM DSX notebook with “Language=Scala” and “Spark version=2.1 OR 2.0″[did not try with other versions]
AND You are trying to access a text file [did not try other formats] that you have uploaded from your personal computer to the IBM Cloud Object Storage


texts: Coding Problem - Extracting values from a column and forming a new dataframe [edited]

texts:
The problem statement requires extracting certain weather parameters for every hour in a particular date as denoted in the dataframe. The column 'hourly' consists of 24 lists in each entry, denoting weather parameters for each hour on that particular date. Is there a way that I can extract the parameter 'CloudCover' for all those 24 hours and form a new dataframe whose columns denote the hours in the day and the corresponding CloudCover value for a single date ?

Edit: Based on the suggestions given below, I tinkered my code and a new problem has come up. While the code suggested by @jahKnows works perfectly fine, It gives only the first value of any parameter in the hourly column entries. For eg. in the date 01-01-2016, the corresponding hourly column entry has 24 values of cloudCover. But the code suggested below gives only the first CloudCover value and moves on to the next date leaving out the other 23 CloudCover values in that particular date. Can you suggest me changes to tackle this problem ? I have attached the updated notebook link and the original dataset link below.

Notebook link(edited): https://anaconda.org/vishwa989796/okayishtrial/notebook

data set link: https://drive.google.com/drive/folders/1wMNOZapHib9AyYFaLdA1jdEdx9DdLnXx


texts: How to tune the hyperparameters of XGBoost and RF?

texts:
How to tune the hyperparameters of XGBoost and RF in python?

There are several methods to tune hyperparameteres of XGBoost and RF such as Bayesian Optimization and meta learning and gridseachcv?
Which one is prefer in which cases?

How Bayesian optimization work for hyperparameteres tuning?


texts: How to use binary text classifier(built using SVM with TF-IDF) to classify new text document?

texts:
I have built binary text classifier using SVM on TF-IDF for news articles(Sports: Non-Sports).

But I not sure how to classify new document using this model. Since TF-IDF is calculated based on the occurrence of a word in all other documents.

Do I have merge test and train data every time I receive a new document for classification? It will change the model as well every time.

Am I missing something? I think, although SVM on TF-IDF giving good results it can not be used in production.

Is there any other way to tackle this issue?

Lets take an example

Training Set:
Doc_1: Chelsea won the match. {Sports}
Doc_2: India won the third test match against Austrailia {Sports}
Doc_3: I want to sleep {Non-Sport}
Doc_4: 13 palace to see in Auckland {Non-Sport}

New Testing Set:
Doc_5: Climate change impacts in Austrailia

Now how can I find IDF score of "Austrailia" in Doc_5 without merging this document with training set?

Since Doc_5 contains the word "Austrailia", it will change the IDF score of "Australia" in Doc_1 will also change, thus model needs retraining


texts:
I'm considering a problem framing within an information retrieval context.

I have a sequence of documents that feature different attributes.  In the web context, these would be webpages.  One attribute could be "is this a top-10 content creator," etc.  When we convert multi-labeled values to binary indicators, we end up with a matrix like:

   a b c d e f
A: 1 0 0 0 1 0
B: 0 1 0 0 1 0
C: 0 0 1 0 0 0
D: 0 1 0 0 0 1
E: 0 0 0 0 1 0
F: 0 0 1 0 0 1

We can ask a user a series of progressive questions about their preferences within this dataset.  Eg. "Do you care about it being from a top 10 content creator?"

However, these are just preferences.  Just because we know the "answer" to a question doesn't immediately invalidate documents with that attribute.  It should just bump it down in the eventual ranking.

The task at hand becomes knowing A) which questions to ask to maximize information gain and B) ranking the resulting list.  This seems like it would be an existing research area but thus far I haven't been able to find anything on it.  Is there a name for this area of algorithm design?


texts: NLP - identifying which adjective describes which noun in a sentence

texts:
I need a method / algorithm for identifying which adjective describes which noun in a sentence.

Sample input:

"The product itself is good however this company has a terrible service"

As an output i'd like to get something like:

[product, good]
[service, terrible]

Could you please point me to some resources / names / algorithms or libraries (no matter which programming language) that might help with solving this?


texts:
So I am running:
spacy train da [several] [options] [here]

And I am getting a nice overview in the console about the performance of each iteration, in terms of P, R and F1-score. I unfortunately closed this window, but would really like to have this condensed comparison of each iter, without having to go through 30 different evals myself, and then manipulating the evals to make it just as compact as the cli.
Is there a cli for outputting these benchmarks again?
Thanks,
Rasmus


texts:
I am reading a book entitled Speech and Language Processing by Daniel Jurafsky and James.
For Acoustic vector, I would like to know why Gaussian assumption is made?
I searched over the internet and could not find the reason.


texts: Explanation of how DeepExplainer works to obtain SHAP values in simple terms

texts:
I have been using DeepExplainer (DE) to obtain the approximate SHAP values for my MLP model. I am following the SHAP Python library.
Now I'd like learn the logic behind DE more. From the relevant paper it is not clear to me how SHAP values are gotten. I see that a background sample set is given and an expected model output is calculated based on this data and the difference is calculated with the current model's output. This difference is the sum of the SHAP values. However, I don't understand how each contribution is obtained? Could you give an explanation with simple terms?


texts: Train autoencoder on 1D data with high and low amplitudes

texts:
I'm using an autoencoder-RNN combination to predict colliding waves in 1D. The training data is obtained by using the 1D Euler equations along with boundary conditions to preduce to waves (left and right) that will collide somewhere in the middle. To get a wide range of training data the boundary conditions for the Euler equations are varied, resulting in different amplitudes and widths of the waves.

The problem here is that some waves have small amplitudes and others have large amplitudes. When train the autoencoder on the normalised data set it will mainly learn how to reproduce the large amplitudes but fails to reproduce or poorly reproduces the small amplitudes.

I'm currently using an MSE loss function to train the autoencoder. Is this the best way to do it for my case or might another loss function perform better?


texts: I want to add demographic data to a data set. Any suggestions on where to find zip code level data?

texts:
Ideally, I would like to add demographic features to my data.  I have zip codes for each observation, so I was hoping to add in the demographic data by zip code.  Unfortunately, I cannot find any demographic data by zip code.

Does anyone have any suggestions on using the zip code tabulation area data for zip codes?

Thanks!


texts:
I am just trying to assign value to a placeholder but i m am getting error:Cannot feed value of shape () for Tensor 'image_input_22:0', which has shape '(1, 51, 51, 64)'

import numpy as np
import tensorflow as tf
res=np.random.random([1,51,51,64])
image=tf.placeholder(tf.float32,shape=[1,51,51,64],name="image_input")
reshaped_image=tf.reshape(image,[1,51,51,64])
init=tf.global_variables_initializer()
res=0
with tf.Session() as sess:
 sess.run(init)
 res=sess.run(reshaped_image,feed_dict={image:res})


texts: How do I interpret the length-scale parameter of the RBF kernel?

texts:
According to the Scikit-Learn documentation for the RBF kernel:
The length scale of the kernel. If a float, an isotropic kernel is used. If an array, an anisotropic kernel is used where each dimension of l defines the length-scale of the respective feature dimension.

I am currently working on a problem where I am setting the length-scale of each individual feature (which I assume is synonymous with dimension here). My understanding is that a smaller length scale implies a more complex function.

My question is, can I use this parameter to explain how well a certain feature will help a model generalize to new data?

For example, if I have a data set which, after optimizing the length-scale value looks like this:
[Feature_1: length-scale = 20]
[Feature_2: length-scale = 1]
[Feature_3: length-scale = 5]

Does this mean that, if I had to pick one feature which would help a model generalize to new data, it would be Feature_1? Is Feature_2 potentially causing my model to overfit? Are these fair assumptions to make?

Note: I am using support vector regression with this kernel.


texts: Splitting a pdf containing batch of scanned documents

texts:
My question is primarily: is there any ML research paper about splitting a pdf containing a batch of scanned documents (eg bank statements) into individual documents?

I have searched for this but I have not found any relevant research paper or any application in general mentioned on the Internet.

I would be primarily interested in the feature engineering of these papers/applications but also in general in the whole approach.


texts:
I am kind of new to NLP and text classification with Convolutional Neural Nets, and I have trained my first models quite recently. I am a little bit concerned with overfitting. I am doing multilabel classification, so my output is a list of 9 numbers (one per label) containing probabilities. I have to set a threshold to the output to get a list of 0s and 1s and assign labels to sentences. When I train the models, I draw a couple of plots to check if there is overfitting or not.

I draw the variation of the F1 score, precision and recall depending on the threshold I set on the output from the model. I see that if I try the trained model with the training data the plots show a perfect F1 score, precision and recall but it shows something much poorer with the test data. In the plots, the orange line corresponds to the training data and the blue one to the test data. Shouldn’t both curves be closer? It seems to me that the training data is perfectly fit whereas the test data is further behind. Is this a good way of checking overfitting? Or is there any other way of checking it?


texts: Stochastic gradient descent and different approaches

texts:
While dealing with stochastic gradient descent, I got confused by several sources which explained it quite differently.

One source explained stochastic gradient descent as standard gradient descent approach but with substantial smaller training set.
The other approach first sums the gradients of all training set and then multiplies with learning rate and updates it. $Q_{t+1}<-Q_t - \tau * \frac{n}{|S|}*\sum_{j\in S}\nabla L_j(Q_t)$

What is the difference? and How does it work with matrices since I take just couple of training examples? (see example below)

Here is an example:
I have sparse matrix M with user x movies where the value in the matrix represents how much the user rates the movie. I decomposed it into matrices P and Q and using stochastic gradient descent I want to optimize P and Q by minimizing the loss function.
Each value of M can be reconstructed by multiplying vector of Q and P.

Let's say I take batches of 10 so it means I take just 10 values from matrix M and respective rows and columns of P and Q which I will optimize.
If I take the second approach there is sum of gradients of all point and then I do the update but how can I sum those vectors since they do not correspond to the same place in matrix Q or P?
In this case, should I then just make new P and Q just with vectors that corresponds to taken values of M? but then it does not make sense because I should optimize the whole Q and P matrices right?

I hope I made it clear enough.


texts: Feature engineering - house price prediction (small dataset)

texts:
I am working on the task of predicting real estate prices. My dataset has only 10 variables described below. I'm thinking about feature engineering but nothing comes to mind.

Variables:

street

city

zip code

rooms

bathrooms

square feet

type

price

latitude

longitude


texts: How are weight updates handled in Batch Gradient Descent vs SGD?

texts:
My current understanding is that in SGD, after each data sample, the loss is used to update each weight.
Ex: With 1000 samples and a network with 10 weights, there will be 10,000 individual weight updates per epoch.

In Gradient Descent and Batch Gradient Descent, how are these updates deferred over multiple data samples? What is being stored at each sample, that can be applied at the end of the batch? Is the loss at each sample averaged over the batch?


texts: Advantage of Centralized training in deep learning over distributed training

texts:
I know distributed training is the ultimate solution for scalability and solving resource constraints and sometimes distributed training outperforms centralized training in terms of accuracy. But I'm really keen to know why general centralized training gives the best accuracy. Though I think, as we don't need to average gradients in each epoch (gradients collected from worker machines in parameter server and parameter server calculates the average gradient and distributes the average among the worker machines) in centralized training, we can consider every gradient, thus can update the weights accurately.

I'm not sure if it is a valid reason, I googled it but got no satisfying answer.


texts: How can I change not only the value of the custom field but the what the field shows itself?

texts: I need to be able to write custom fields in Power BI but the values of the fields change based off of company. For example, Twitter vs Instagram - field 1 for Twitter should show company name and field 1 for Instagram should show company revenue. So not only does the field change but the value of the field changes too. I have an excel sheet that lists around 30 custom fields for 2 different companies so I was wondering what the easiest way around that would be. Thanks!


texts: Methods for standardizing / normalizing different rank scales

texts:
I know there is the normal subtract the mean and divide by the standard deviation for standardizing your data, but I'm interested to know if there are more appropriate methods for this kind of discrete data. Consider the following case.

I have 5 items that have been ranked by customers. First 2 items were ranked on a 1-10 scale. Others are 1-100 and 1-5. To transform everything to a 1 to 10 scale, is there another method better suited for this case?

If the data has a central tendency, then the standard would work fine, but what about when you have more of a halo effect, or some more exponential distribution?


texts: Non-Convex Constraints for Classification Problems

texts:
I am willing to create a hypothetical non-convex constraints for the purpose of practising nonlinear classification using an algorithm. I thought of such constraints in the form: $x^TAx + Bx \leq c$.

I am curious if this qualifies for non-convex constraints, and if the matrices $A$ and $B$ are necessarily PSD. Or could I possibly have more than this constraints?

I would like if someone explains this or refers me to any text/paper I could read up. My Mathematics is kind of rusty at the moment.


texts: Overfitting while fine-tuning pre-trained transformer

texts:
Pretrained transformers (GPT2, Bert, XLNET) is popular and useful because of their transfer learning capabilities.
Just to remind: The goal of Transfer learning is is to transfer knowledge gained from one domain/task and use that transfer/use that knowledge to solve some related tasks. This is done by training a model on a huge amount of labelled data (which we already have and probably easy to get), then remove last few layers and fine-tune the model for the new related task with task-related dataset.
I took a recent pretrained transformer published by Google called XLNET, and just add classification layer of the top of that and fine-tune the whole network. (Which is the main intention of this kind of model, correct me if I am wrong )
Problem is, the model is hugely overfitting. I have 1200 example to train and each has 350 words on average.
To overcome the overfitting, I set the dropout of each layer of the transformer from 0.1 to 0.5. This did not work. So I  decreased the number of trainable parameters (since the transformer has a huge number of parameters), by freezing first 10 layers (11 layers + 1 classification layer in total). Even that does not work. So I counted the number of trainable parameters in the last layer. There are 7680000 parameters which are very high compared to my dataset (around 1200*350= 420000 words). So, this high number of tunable parameters is the most possible reason for overfitting.
Here is the loss graph:

My questions are:
Do you see any flaw in my analysis?
Is there anything I can do to decrease overfitting? (tried with low learning rate and large batch size)
If my analysis is correct, then the claim that "fine-tune pre-trained transformers with small dataset" is bit misleading and datasets should not be so small. Am I correct?


texts: Adding layers in the middle of the deep learning model

texts: I want to add new layers to a complex CNN model in Keras, how can I add a new layers in the middle of the sequential deep learning model?


texts: Convolutional neural networks for non-image applications?

texts:
If I remember correctly, Convolutional neural networks (CNN) have first been developed for image classification purposes (see work from LeCun et al.).

The convolution process that "slides" over the input image is helpful indeed to detect some features.

Question: do you have examples of successful application of CNN for non-image data?


texts: How to simulate customer walk-ins for a given period in a fast food chain in R

texts:
I would really appreciate it if someone could tell me where I would start in tackling the following tasks in R.

I'm not a data-science expert and am trying to teach myself data-analysis from a background in physics. I have introductory statistics and have just finished Udacity's course in R programming and exploratory data analysis.

Tasks:

1 )Simulate customer walk-ins for a given period. The simulations should take into account peak volumes at different times of the days and differences in week days and weekends.

2) Simulate customer purchases based on the customer walk-ins. Make assumptions on the average dollar price and variations for different times of the day.

Thank you


texts: Getting worse results after Hyperparameter Tuning(Grid/Random Search/TPOT)

texts: I have a problem with Hyperparameter Tuning. Usually I getting almost the same results(or worse) than before tuning. Usually default parameters of classificator(regressor) give me a best score. Anybody could recommend other techniques or give me a tip what I should improve?


texts: How to make output dimensions match input dimensions in CNN?

texts:
I have training samples of the following shape: (1000,2). These are numeric sequences, each of length = 1000, dimensions = 2.

I need to build a Convolutional Neural Network to output Predictions/Sequences of the same shape (1000, 2). From what I understand, after applying convolution and pooling, the height and width of the input is reduced.

How should I then set up the fully connected layer(s) and an output layer in my CNN, so that the output dimensions match the input dimensions? Or in general, how should I set up my CNN architecture to achieve this?


texts:
"Machine Learning over JSON" describes some issues surrounding the classification of JSON documents. Namely,

Categorical Features
Data is Hierarchical
Missingness is Chunky

The first two have fairly standard solutions by flattening the JSON and things like one-hot-encoding, vectorizers, etc..

The later is where I am stuck. I have thought about,

Binary flags for the feature, but this limits the number of features
Sequence approaches, but it's unclear how to handle nested data structures

What are approaches I can use to handle the variable number of features?


texts:
I am using the seaborn violin plot feature however the size of the violins aren't what I would expect them to be. Does anyone know whether this is something I am doing wrong, or as I suspect whether this is a bug in seaborn.

The code I am using is the following;

 sns.violinplot(x = "Issue Type", y = "issue_adj_risk_score", data=subset, scale="count", cut =0)

and the chart I can see is like this

As I understand it, becuase I have used the scale ="count" parameter, the sizes of each of these violins should be proportional to an overall count of the number of things in each area. In 'category 1' however (pink) I have a total of 936 with an issue_adj_risk_score of 1 whilst in category 2 (orange) I have a total of 1212 with an issue_adj_risk_score of 1.

I am confused however as the pink is larger than the orange! :-(

Does anyone know why this might be?

EDIT

I am now seeing further problems where the range of a particular category is inconsistent. I have a dataset and used violinplot to plot the below

Here we can see that a certain category (which I have highlighted in red) has a range from 1-4. This surprised me so I looked into the underlying data and it doesn't contain any values at 4. I then took a subset that contained this category only and plotted a violinplot with the same data and got this

Which is what I would have expected to see in the initial chart!

To investigate this further I plotted a boxplot based on the original data once again, and got the following

Which is consistent with the first chart but doesn't reflect the actual data. How can all of these 3 charts come from the same data!!??


texts: How to evaluate clustering success in a completely unsupervised system?

texts:
The algorithm in question is Kohonen's SOM. But the question could also apply to PCA and some others.

When the umatrix (or the codebook?) is examined, is there a way to tell how successful clustering was?

And would it be a good idea to apply GA's to optimize size, lattice structure, learning rate, and the learning degradation functions as well as the epoch count for clustering, or is there a danger of overfitting in this instance?

Assume that the SOM data is coming from a demonstrably weak PRNG and that the very first attempt shows a distinct structure. Is there some statistical property or algorithm that can evaluate the presence and degree of structure to be used for a GA fitness function?


texts: Training an RNN with examples of different lengths in Keras

texts:
I am trying to get started learning about RNNs and I'm using Keras. I understand the basic premise of vanilla RNN and LSTM layers, but I'm having trouble understanding a certain technical point for training.

In the keras documentation, it says the input to an RNN layer must have shape (batch_size, timesteps, input_dim). This suggests that all the training examples have a fixed sequence length, namely timesteps.

But this is not especially typical, is it? I might want to have the RNN operate on sentences of varying lengths. When I train it on some corpus, I will feed it batches of sentences, all of different lengths.

I suppose the obvious thing to do would be to find the max length of any sequence in the training set and zero pad it. But then does that mean I can't make predictions at test time with input length greater than that?

This is a question about Keras's particular implementation, I suppose, but I'm also asking for what people typically do when faced with this kind of a problem in general.


texts: calculating average of each session by unique itemID

texts:
i have a data frame

 > data
session id        item id
  1                 1
  1                 1
  1                 1
  2                 1
  2                 2
  3                 1
  3                 0
  3                 3
  3                 2

and i want to calculate avg of each session id like this uniqueitemsinsession/totalitems= ans (1/3=0.333)and get result like this

>  result
session id       avg
   1             0.333
   2             1.000
   3             1.000

how can i do this??


texts: Logistic regression vs Random Forest on imbalanced data set

texts:
I have an imbalanced data set where positives are just 10% of the whole sample. I am using logistic regression and random forest for classification. While comparing the results of these models, I have found that the probability output of logistic regression ranges between [0,1] while that of random forest ranges between [0, 0.6].
I cannot share the data set but my doubt is around the working of these algorithms. How can random forest generate probability less than 0.6?


texts:
I was looking at the source codes of MinMaxScaler on Github. I know that when you fit a preprocessing class to a dataset, it takes the data and prepares it for transformation.
Let's say, I fitted MinMaxScaler to X_train and transformed it. But how does transform work when I use another dataset, let's say X_test? When you call transform(), does it replace the datasets in the use?


texts: Multiclass classification with many classes and wide range of sample sizes

texts:
I'm working on a free text classification problem with over 100 classes in the training data. There is huge variation in the sample sizes of the classes: ranging from 1 to around 6000. I am using a custom-made algorithm which builds a training model of each class's most significant keyword or phrase (up to 3 consecutive words). The significance is a value between 0 and 1 and is the difference in the frequency that the term occurs in the class and the frequency that it occurs among all data not in the class. I'm then using this model to test whether whether new, manually class-labelled records have the correct class by assigning PASS if the new record's class has a significance level > .9 AND the test record's free text contains the keyword/phrase. If its > .9 and lacks the keyword, I'm giving it a FAIL. And if the class significance is <= .9, then I'm giving it a NOT CHECKED, regardless of if the test text contains the keyword or not because the strength of the top keyword is too weakly associated with the class.

I developed this custom-made algorithm after finding that Naive Bayes (a slightly more complicated probabilistic method taking into account all words) wasn't doing that well...I think because of the nature of my data: the text is often short and there are some classes where 100% should contain a specific keyword.

The results so far are promising...far fewer false positives than Naive Bayes. But I've found that classes with tiny sample sizes in the training data had a LOT more false positives than with large sample sizes.

So my question is: what would the best way to account for % confidence based on the sample size?

The best idea I have right now is to use the Sigmoid function, where input x = sample size and output y = something like % confidence, then multiple this Sigmoid output by the significance level described above and only doing the incoming record class check if this modified significance level is > .9. Are there better options?


texts: How does joint probability generate new data in generative model

texts:
Learning about different Machine Learning concepts, I came across Generative and Discriminative model. To infer from what I have studied, generative model is based on P(x,y)(Joint probability distribution) whereas discriminative model is based on P(y|x) which is conditional probability. But the question is why; infact HOW? How does joint probability give rise to new data meanwhile conditional probability just works on current dataset?
Can anyone explain me with an intuitive example or maybe some link which would elucidate the concept ?


texts: Why are the weights of my first layer and last layer in the CNN change while the middle layers don't?

texts:
The weights of my first and last convolution layers do change in a noticeable way. However, the rest of my convolution layers, in the middle, do not.

I should add that all convolution layers' biases change noticeably, however the kernel weights of the middle layers (as opposed to the first and last layers) do not change significantly.

Has anyone else encountered this?

Can anyone suggest why this is might be happening?

Can I rule out vanishing gradients...?

Any ideas for resolution (if this is at all a problem)?

Thanks in advance.

I'm happy to add details based on comments.


texts: Latent feature extraction using dnn and word2vec embeddings

texts:
I recently read a journal about a tag-aware recommender system. There is a part in the paper which I do not understand.

They used word2vec first and using embeddings as input to a DNN to extract latent features. I'm not sure what could the latent features be. Could anyone give me some hints?


texts: Predicting Composition of Chemical Compounds

texts:
I have a dataset which has names of compounds and their compositions. Like below

Sulphuric Acid=>[H,S,O] (Hydrogen, sulphur, oxygen)
Oxalic Acid=>[H,C,O]
Sodium Oxalate=>[Na,C,O]
Potassium Sulphate=>[K,S,O]
...

Now I would need to train a model which can tell me Sodium Sulphate as [Na,S,O]. Note that sodium sulphate represents something not in training dataset.
I have tried searching for possible ideas but nothing came up. Then I thought it could be hierarchical clustering like

 Sodium-Oxalate
    |    |
Oxalate  Sodium
|  |      |
C  O      Na

But then in hierarchical clustering the base/leafs are different. But here they are shared. Its like a graph. So What machine learning algorithm can help? Any other clustering? NLP/word clustering (if yes How)?

Another approach I could think of is like Word2Vec where I generate embeddings for each word (C,H,Sodium), all will have embeddings. And based on what is closer to the word I am asking I will give output. But this needs huge amount of data. I only have around 1k common compounds. And the approach won't generalize to any problem like this with less data.


texts: Decision trees vs Oblique decision trees

texts:
What are Oblique decision trees ?
What are the differences between it and classic decision trees ?


texts:
I've used optical character recognition (OCR) on a historic directory, and am trying to clean up the text with Microsoft Word. Specifically, I need some help writing a Regular Expression to combine two lines together. For example something that is

John Smith, 87 Bank

Bldg

should actually be

John Smith, 87 Bank Bldg

I've tried several approaches, but haven't been successful at all. Can anyone help me with this?


texts: Given a query image Q and two other images X and Y, how to determine which one is most similar to Q?

texts:
Given a query image Q and two other images X and Y (you can assume they have more or less the same resolutions if that simplifies the problem), which algorithm would perform extremely well at determining which image between X and Y is most similar to Q, even when the differences are rather subtle? For example, a trivial case would be:

Q = image of mountains, X = image of mountains, Y = image of dogs, therefore it is clear that sim(Q,X) > sim(Q,Y).

However, examples of trickier cases would be:

Q = image of a yellow car, X = image of a red car, Y = image of a yellow car, therefore sim(Q,Y) > sim(Q,X) (assuming the car shapes are more or less the same).
Q = image of a man standing up in the middle with a black background, X = image of another man standing up in the middle with a black background, Y = image of a woman standing up in the middle with a black background, therefore sim(Q,X) > sim(Q,Y).

Which algorithm (or combination of algorithms) would be robust enough to handle even the tricky cases with very high accuracy?


texts: IDE for Machine Learning workflow - AWS SageMaker

texts:
Looks like we don't really use an IDE in any of Machine Learning workflow stages if we use AWS SageMaker.
Entire work is done in jupyter notebook. Is this correct?


texts: Does Double Median Absolute Deviation work for every distribution for the purpose of outlier detection?

texts:
I am using Double Median Absolute Deviation for finding outliers in a 1-D data. As mean with standard deviation gets influenced easily by outliers, that's why I chose median based approach. And the reason for choosing Double MAD over MAD was that MAD doesn't give appreciable results when gets a asymmetric distribution. I also want that the algorithm should be pessimistic in classifying points as outliers i.e. the algorithm must set tighter limits.
So I have 2 questions basically-

Is MAD (or Double MAD) good for this purpose of outlier detection? Or is any better approach(it can be statistical or any clustering/density/distance based algorithm as well)?
There must be some distributions(or corner cases) for which MAD (or Double MAD) will fail. If possible, I want to know such cases.


texts: Is it acceptable to select a random child node when using a Decision Tree (trained via ID3) to predict if an unknown attribute value is encountered

texts:
I am writing a decision tree trained with the ID3 algorithm from scratch. I wanted to be able to train on and classify continuous data, so I implemented k-means clustering and reduced the range of values of any input training or predicting data.

However, I ran into the problem where an attribute value that was not encountered during training, in a deep node somewhere in the tree, but that exists further up was encountered. All data points with this specific attribute value probably ended up on a different branch of the tree.

So to 'solve' this, whenever an unknown attribute value is encountered for a node, I sent it down a random existing branch.

I get 95.45% accuracy using randomly split 85%-15% training-test data with iris.

Is this an acceptable approach to take or have I gotten something wrong here?

Here's the code: https://github.com/jamalmoir/ml_components/blob/master/ml_components/models/decision_tree.py

Thanks


texts: Tensorflow model works for classification but not for regression (all predictions equal the output layer bias)

texts:
I'm trying to build a model for FX prediction.  It's giving some promising results for classifying each period as buy/sell/neutral.  When used as a classifier, actual returns are converted to 0, 1, or 2 to represent the possible actions (sell=0, neutral=1, buy=2) based on whether the future price change exceeds a threshold to the high or low side.

However, when trying to predict the future change as a linear regression problem, it's making nonsensical predictions.  Every time I train the model, it quickly converges to a different fixed value prediction for every period.

Here's a boiled down version of the model build/compilation:

def build(self, is_evolution=False):
    self.model = models.Sequential()
    self.model.add(layers.Dense(self.num_layers, activation="relu", input_shape=(self.x_train.shape[1],)))
    self.model.add(layers.Dense(self.num_layers, activation="relu"))
    if self.is_classification:
        class_counts = np.array(list(map(lambda x: self.y_train[self.y_train == x].shape[0], self.classes)))
        total_count = self.y_train.shape[0]
        bias_initializer = tf.constant_initializer(tuple(class_counts / total_count))
        self.model.add(layers.Dense(len(self.classes), activation="sigmoid", bias_initializer=bias_initializer))
    else:
        self.model.add(layers.Dense(1))

    loss = "sparse_categorical_crossentropy" if self.is_classification else tf.keras.losses.MeanSquaredError()
    self.model.compile(optimizer=tf.keras.optimizers.Adam(), loss=loss, metrics=["accuracy"])

Here's an example of the predictions when is_classification is set to False.  Not only is this nonsensical because all of the predictions are the same, but the numbers in this case are also very big (i.e., the actual mean is very close to zero), although the prediction numbers can vary quite a bit with different trainings.

[[0.00081437]
 [0.00081437]
 [0.00081437]
 ...
 [0.00081437]
 [0.00081437]
 [0.00081437]]

Based on suggestions here: Tensorflow regression model giving same prediction every time, I've tried changing the learning rate to no avail.  I'm also making sure to scale the train and test set, although I'm not sure that should matter for linear regression.

At first, I thought this was just because the dataset was very noisy, but thinking about it more, I think a model that can classify a buy or sell should also be able to make regression predictions when the output layer is modified as it is above.

Any thoughts are appreciated.

Update: I was inspecting individual weights and I see that the linear regression predictions are equal to the output layer bias, however, the other weights appear to be "normal" (i.e., various positive and negative weights).


texts: How to read html tables under multiple headers and combine them in a single pandas dataframe?

texts:
I have an html file like this:

Group 1

    Col1
    Col2
    Col3

    ValA
    ValB
    ValC

Group 2

    Col1
    Col2
    Col3

    ValP
    ValQ
    ValR

I want to read it into Pandas as if it had this structure:

   Caption
   Col1
   Col2
   Col3

   Group 1
   ValA
   ValB
   ValC

   Group 2
   ValP
   ValQ
   ValR

I can do it easily with PowerQuery the language for PowerBI:

let
    Source = Web.Page(File.Contents("multiple_tables.html")),
    #"Expanded Data" = Table.ExpandTableColumn(Source, "Data", {"Column1", "Column2", "Column3"}, {"Col1", "Col2", "Col3"}),
    #"Filtered Rows" = Table.SelectRows(#"Expanded Data", each ([Caption] <> "Document") and ([Col1] <> "Col1"))
in
    #"Filtered Rows"

Is there a way to achieve this effect in less than 10 lines of code using Python/Pandas plus some html parser opensource library? Or should I resign myself to writing lower level code to handle this?


texts: How to find the count of consecutive same string values in a pandas dataframe?

texts:
Assume that we have the following pandas dataframe:

df = pd.DataFrame({'col1':['A>G','C>T','C>T','G>T','C>T', 'A>G','A>G','A>G'],'col2':['TCT','ACA','TCA','TCA','GCT', 'ACT','CTG','ATG'], 'start':[1000,2000,3000,4000,5000,6000,10000,20000]})

input:
 col1 col2  start
0  A>G  TCT   1000
1  C>T  ACA   2000
2  C>T  TCA   3000
3  G>T  TCA   4000
4  C>T  GCT   5000
5  A>G  ACT   6000
6  A>G  CTG  10000
7  A>G  ATG  20000
8  C>A  TCT  10000
9  C>T  ACA   2000
10 C>T  TCA   3000
11 C>T  TCA   4000

What I want to get is the number of consecutive values in col1 and length of these consecutive values and the difference between the last element's start and first element's start:

output:
 type length  diff
0  C>T  2   1000
1  A>G  3   14000
2  C>T  3   2000


texts: how can i fix this problem ? I tried to train a simple linear VSM classifier

texts:
library(kernlab)
 letter_classifier <- ksvm(letter ~ ., data = letters_train,
                       kernel = "vanilladot")

>  Setting default kernel parameters   Error in .local(x, ...) :    No
> Support Vectors found. You may want to change your parameters In
> addition: Warning message: In .local(x, ...) : NAs introduced by
> coercion


texts: How can i build a machine learning model to identify a specific word?

texts: I am trying to build a model that can be used to identify the word sales whenever a group of text are passed through the model. Should I use Azure or Python for this? I have done the text preprocessing and extracted the n-gram features in Azure, what should I do next and which model should I use?


texts: Why do you need to learn statistics for machine learning?

texts:
This maybe a very broad question. Please do not delete.

However, I am curious on why do we need to learn statistics for Machine Learning?

This is given that the software libraries used already have the ‘knowledge’ encoded.
Of course, a general understanding of how things work under the hood, but other than that is there a need for a more in-depth understanding?

Same question applies regarding inner workings of various models. How will a good understanding (including the maths) help?


texts:
I have read a couple of documents that explain in detail about the greater edge that GPT-3(Generative Pre-trained Transformer-3) has over BERT(Bidirectional Encoder Representation from Transformers). So am curious to know whether BERT scores better than GPT-3 in any particular area of NLP?
It's quite interesting to note that OpenAI's GPT-3 is not open-sourced whereas tech behemoth Google's BERT is open-sourced. I felt OpenAI's stance and the hefty price tag for GPT-3 api is in stark contrast to its mission statement(OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity).
https://analyticsindiamag.com/gpt-3-vs-bert-for-nlp-tasks/
https://thenextweb.com/neural/2020/07/23/openais-new-gpt-3-language-explained-in-under-3-minutes-syndication/
https://medium.com/towards-artificial-intelligence/gpt-3-from-openai-is-here-and-its-a-monster-f0ab164ea2f8


texts:
I am having some issues doing operations on my data table in Orange.
I need to split my table into two different tables, being a table containing all the instances for y = yes and a table containing all the instances for y = no. How can I most easily do this?
Thank you in advance
Emma


texts: Multivariable linear gradient descent resulting in inf

texts:
I am trying to implement a multivariable gradient descent algorithm, it seems to start working fine, and works on smaller datasets, but applying it to larger datasets the variables overflow and cause an inf (at least thats what I think is happening).
def standard_fit(self, x, y):
    # add a column of 1s
    x = np.array(np.hstack((np.ones((len(x), 1)), x)))
    dt = np.zeros(self.m)

    for i in range(self.max_iter):
        # Predict y
        pred_y = np.dot(x, self.theta)

        # Calculate loss
        loss = pred_y - y

        # Calculate gradient of the cost
        dt = np.dot(x.T, loss)

        # modify theta
        prev_theta = self.theta.copy()
        self.theta -= self.alpha * dt

        # Check precision
        if abs(sum(self.theta - prev_theta)) < self.precision:
            print("PRECISION MET!")
            break

        if i % 100 == 0:
            # Calculate cost
            pred_cost = self.cost(pred_y, y)
            print("ITERATION " + str(i) + ": " + str(pred_cost)

m is the number of features (+1 to consider the row of ones).
precision is how small a step it should take before stopping
theta is my feature weights

I've been stuck on this problem for more time than I'd be willing to admit. Watching videos on it I see people dividing the gradient by dividing it by the number of features after the calculation, this does nothing.
I also saw someone sum the gradient, which really makes no sense to me as surely then it would modify all the thetas evenly?


texts:
I have the following data set

df <- data.frame(student=c(1,2,3,4,5,6,7,8,9), sat=c(365,0,545,630,385,410,0,655,0), act=c(28,20,0,0,16,17,35,29,21))

student     sat    act
    1       365    28
    2         0    20
    3       545     0
    4       630     0
    5       385    16
    6       410    17
    7         0    35
    8       655    29
    9         0    21

and I'd like to create a new field with the following conditions

If there is an SAT score > 0 use SAT score
If SAT=0, then convert the ACT to an SAT score using the rubric here. (When there was a range in the SAT score, I just used the median.

ACT SAT
8   200
9   210
10  220
11  225
12  250
13  285
14  325
15  360
16  385
17  410
18  440
19  465
20  485
21  505
22  525
23  545
24  560
25  575
26  595
27  615
28  635
29  655
30  675
31  700
32  725
33  750
34  775
35  790
36  800

This is one heck of an ifelse statement. I've tried this:

df$newgrade=-ifelse(ACT=8,200, ifelse (ACT=9,210, ifelse(ACT=10,220, ifelse (ACT=11,225, ACT=12,250, ifelse(ACT=13,285, ifelse (ACT=14,325, ACT=15,D, ifelse(ACT=16,C, ifelse (ACT=17,B, ACT=18,D, ifelse(ACT=19,C, ifelse (ACT=20,B, ACT=21,D, ifelse(ACT=22,C, ifelse (ACT=23,B, ACT=24,D, ifelse(ACT=25,C, ifelse (ACT=26,B, ACT=27,D, ifelse(ACT=28,C, ifelse (ACT=29,B, ACT=30,D, ifelse(ACT=31,C, ifelse (ACT=32,B, ACT=33,D, ifelse(ACT=34,C, ifelse (ACT=35,B, ACT=36,D))))))))))))))))))))

I tried to follow the example at the bottom of this page but it didn't work.

Someone else on another board suggested:

df$newgrade<-ifelse(df$sat == 0, conversion$SAT[match(df$act, conversion$ACT)], df$sat)

but then a new issue presented itself:
If there is neither an ACT nor a SAT score. How can it put a 0 in for that group??

Thank you for any assistance you may bring.


texts: How to train neural networks with large sized data sets?

texts:
I have a dataset size of ~500000 with input dimension 46.
I am trying to use Pybrain to train the network but the training is extremely slow for the whole dataset. Using batches of 50000 data points, each batch takes more than 2 hours for training.

What are caveats of optimizing the network design so that the training is faster?


texts: Time series normalization using min max technique

texts:
I have a time series dataset and I would like to normalize the data (diff which is of type list) as below using Min Max technique. But, I get the following error:

Code:

# split data into train and test-sets
train, test = diff[0:1486], diff[1486:2123]
from sklearn.preprocessing import MinMaxScaler
# scale train and test data to [-1, 1]
def scale(train, test):
    # fit scaler
    scaler = MinMaxScaler(feature_range=(-1, 1))
    scaler = scaler.fit(train)
    # transform train
    train = train.reshape(train.shape[0], train.shape[1])
    train_scaled = scaler.transform(train)
    # transform test
    test = test.reshape(test.shape[0], test.shape[1])
    test_scaled = scaler.transform(test)
    return scaler, train_scaled, test_scaled
# transform the scale of the data
scaler, train_scaled, test_scaled = scale(train, test)

Error:

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.


texts: Trying to get Orange connected to Postgres Database

texts:
I recently came across Orange, and am very keen to try it for our company's purposes (around 20 potential users, perhaps more). I have installed Orange (latest version) and am trying to connect to our database, as it has a large (multi GB) database of telemetry collected from the sawmill.

However I am unable to connect. I have installed psycopg2 as instructed, but when I try to connect, I get a message saying "Could not open Extension file" ... Apparently a tsm_system_time.control file is missing. And it indeed is not in the folder as directed by psycopg2, I have looked.

I have an IT specialist friend, who tried installing Orange on his laptop (he also has a postgres database on it) and he got the same message, except he apparently was missing a quantile.control file. He didn't proceed further, but I am, wondering

What are all these "missing" .control files, and
Why does psycopg2 need them if they are not pertinent to database operation (everything else works in the database, including reporting etc.)
How do I resolve the problem? No help forthcoming online in terms of missing .control files.

I'm running on Windows 7 (Database is a local server running Windows 2012 Server)


texts: Use of Nash-Equilibrium in big data environments

texts:
Has there been a successful implementation of Nash-Equilibrium in big data problems like suggesting a best buy in a stock market, in traffic monitoring systems or crowd control systems.
All the above mentioned scenarios have competitive environments and one needs to get the best possible solution in them, which should suit well for Nash-Equilibrium cases.


texts: Why do we need the hyperparameters beta and alpha in LDA?

texts:
I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind:

First: Why do we need to add alpha and gamma every time we sample the equation below? What if we delete the alpha and gamma from the equation? Would it still be possible to get the result?

Second: In LDA, we randomly assign a topic to every word in the document. Then, we try to optimize the topic by observing the data. Where is the part which is related to posterior inference in the equation above?


texts: How to search for an optimal dithering pattern?

texts:
I'm trying to find an optimal dithering pattern which can be used as a threshold on a greyscale image to generate a 1 bit black and white image. Ideally it would be optimal in the sense that a human would judge it perceptually most close to the source image.

For instance, dithering with white noise looks pretty bad:

While dithering with blue noise looks a lot better, despite having the same amount of error:

(images from https://blog.demofox.org/2017/10/31/animating-noise-for-integration-over-time/)

While white noise doesn't look good, it can be easily generated in real time, whereas blue noise cannot.

Because of this, I'm trying to find a dithering pattern which is:

NxN in size
Has an even distribution of values: 0/N, 1/N ... (N-1)/N
Perceptually the best bang for the buck

After I have the best pattern for a given N (or a reasonably good pattern), step 2 would be to try and figure out how to calculate it mathematically, but the first step is to find the pattern.

Does anyone have any recommendations on a good way to do this?

What I'm currently trying is using simulated annealing to find an optimal solution.  I'm using SSIM to calculate the score of how good a solution is, for use in in the simulated annealing logic.

For the simulated annealing, I initialize the pattern by doing this:

Initialize the NxN pattern such that row M is set to M/N.
Shuffle the pattern

When calculating a possible move to make for the simulated annealing, I pick two values at random in the NxN pattern and swap them.

This isn't working very well for me, but I am unsure if this is not a good way to go about it, or if maybe I'm choosing bad parameters for the simulating annealing and/or the SSIM.

The simulated annealing has a starting temperature, and a specified number of iterations to take it from there down to temperature 0.

SSIM on the other hand has c1, c2, and c3 which I'm setting to zero.  I am also doing SSIM of every 8x8 block in the dithered image (ALL of them, even though they overlap) and averaging them together to get the SSIM score.

Here is an example of a 5x5 dither pattern this algorithm came up with, which really doesn't look good at all.

I manually derived a 5x5 by hand which looks much better in fact:

Does anyone have any recommendations on how I might search / solve this problem better?


texts:
In order to install some libraries, we just use:

conda install pandas

While in other cases we use:

conda install -c conda-forge html5lib

What is the difference between them?


texts: How does Stanford CRF encode NER string features?

texts:
Most features created by the NERFeatureFactory are strings e.g. from usePrev, useNext, useNGrams etc. From my understanding, that's too many tokens to fit in a dictionary or to use embeddings. I don't see how the UNKNOWN embedding would bring any value given that most features are not known words. I've been looking at the code on Github but haven't figured it out yet.

I love New York! > love > love-I-W-PW, love-New-W-NW, #lo#, #ov#, #ve# etc


texts: How to create a heatmap inside a polar chart

texts: I would like to build something like this. Creating a polar chart isn´t an issue, but i have no idea how to implement the round areas and the color gradients into the plot.


texts: An epoch step is taking too much time on GPU!

texts:
I am trying to train a CNN on phoneme recognition. My dataset is composed up of 50 Million sample, each sample has 120 feature (signal parameters). My input's shape is (11,120) and the target is a phoneme class.

The CNN architecture is as follows:

CONV (32 activation maps / ReLU activation) - POOL (max pooling)
CONV (64 activation maps / ReLU activation) - POOL (max pooling)
3 DENSE layers (each contains 1024 neurons / ReLU activation function / dropout =0.5)

The output is a softmax layer for 36 class.

I used the fit_generator as I generate my data batch-by-batch. My batch size = 512. I tried to train the model on 10 Million of sample = almost 20K of batches however an epoch took me more than 3 hours.

The server i'm using has 2 GPUs:

Is that normal with such GPU characteristics? If yes, what can I do to accelerate my model training phase? Especially since I would like to train it on the whole dataset?


texts: Caffe net.predict() , predict same probability

texts:
I trained pre-trained ResNet-101 following caffe model(By Kaiming He github) with 800000 data for training and 200000 data for validation. After I train this model, I got 59% accuracy for 1st and accuracy-top5 is 82% with 30 epoch as seen below picture.

But when I tried predict some images with this model(net.forward()), the results always produce same probability like below even though I tried with other images.

First thing I thought was image preprocessing problem in predict step like subtracting mean values or adequating batch size corresponding with training step. But all of these step was correctly set up.
I checked all other questions having a same problem with me but couldn't find a solution. As the above picture(1) is showing, I assume the training process wasn't something wrong.

I followed "Image Classification and Filter Visualization(00-classification.ipynb)" file provided from Caffe editing model_def and model_weights with my model_def and model_weights having a 30 epoch.


texts:
Currently making a bot that condenses news articles. I'm tagging sentences as important or not important using a simple BERT classifier. The results were... not great. I'm really interested in how I can improve the results using LSTM.

I'm now batching 5 sentences together, calculating their BERT encodings, and then using 2 LSTM Layers, one backwards one forwards, to predict if the sentence is important.

Unfortunately I'm now calculating 5 times the number of embeddings, and if it doesn't work, I can't seem to figure out how to feed a variable number of things into BERT using Tensorflow, to see if I can tweak some results.

Are there other methods to add surrounding sentences to this context?


texts:
I am trying to build multi agent reinforcement learning in 2D grid environment(Combat & Switch) using OpenAI. When I am trying to build game using ten agents, five opponents and the dimension is(15*15) using DDPG algorithm.
So far, I looked into deep reinforce algorithm. However, am not clear why the output of the game is disappearing , such that it allows for modifying entries of the matrix.


texts: Add extra term weight when grouping strings by similarity?

texts:
I'd like to convert a set ducoments to term weight matrix(features) by tf-idf.
Then calculate the similarity of two document by their features.

similarity is compute by result = matrix*matrix.T > 0.9 detail here, and group them by the result(loop result, if result[1,0] >0.9, then index 1 is similar with index 2)

Now I have a lot of resource to group.

For example, I have some books with different names(more complex in fact)
I can roughly group these books name by similarity, like below:

step1:

group 1:
    1.The Three Body Problem vol 1
    2.[Chinese]The Three Body Problem no 1
    3.The Three Body Problem 2
    4.The Three Body Problem vol 3[Japanese]
    5.Problem of Three Body vol 3
    6.(xx)The Three Body Problem 2
    7.The Three Body Problem 1[English]
group 2:
    1.Another book 1
    ....

But xxx 2 and xxx vol3 is needless when I want to find xxx vol 1 ,
So have to do

step2:
   tokenize each book name again, use some patterns/rules to extract the book number to distinguish them.

Is there any way to add some term(such as Arabic numerals:0-1, English number:one- twenty) with high dissimilarity weight, to make step1 result

group 1:
    1.The Three Body Problem vol 1
    2.[Chinese]The Three Body Problem no 1
    7.The Three Body Problem 1[English]
group 2:
    3.The Three Body Problem 2
    6.(xx)The Three Body Problem 2
group 3:
    4.The Three Body Problem vol 3[Japanese]
    5.Problem of Three Body vol 3
group 4:
    another book 1
    [xx]another book vol.1

update

If there are some titles like below:

1. There are 2 man vol.1
2. (xx)There man 2 boy 2

I need add a lot detection(number position or something else), that's why I want a way to add a extra weight to somewhere(to avoid step 2, the redundant tokennizing and custom extraction rule).

I think the similarity weight may work like:

tfidf weight two title, plus each number weight, then calculate the similarity matrix.
But now I am using tfidf matrix power to get similarity matrix,
I don't know how to add a extra weight to  tfidf weight result, the weight meaning is different between tfidf weight and what mentioned extra weight.

I want to know where and how to add the proper extra weight, how calculate it value?


texts: Which are the Best Big Data Services companies in 2018

texts: are there any companies with a proven track record to help in big data services or big data consulting services.


texts: Multi-classification: low precision due to imbalanced classes in test data - what to do?

texts:
I built a multi-classification model with 3 result classes (XGBoost using R's caret-package): A, B and C. I undersampled my training data - so every class is equally abundant for training. The following numbers relate to test data.

My recall numbers are pretty good: rec(A) = 71%; rec(B) = 83%; rec(C) = 78%

For a balanced distribution also my precision is pretty good: prec(A) = 76%; prec(B) = 74%; prec(C) = 79%.

Unfortunately, in real life class C is much more abundant than classes A and B (76% to 14 % to 10%). Therefore, when applying test data with realistically imbalanced result classe I obtain the following precision values: prec2(A) = 50%; prec2(B) = 43%; prec2(C) = 96%.

Apparently, just by chance, whenever I estimate class A or B the chances are pretty high that it is in fact class C.

Ideally, I would like to have high precision also when facing imbalanced data. What would you do when you were in my shoes?

I thought of the following things:

whenever classes A and B are estimated run the data through other models in order to verify the estimate (ensemble)
change the training metric; by default caret uses accuracy - I could try using a macro-f1 score
keep the class imbalances in the training data and train on a precision metric


texts:
I am trying to build a RNN that can take time series as input for example from various sensors.
While building the RNN there are a couple of initial lengths that must be defined.
The size of the cell and the length of how much you will unroll the RNN.

From what i understand, cell size is arbitrary and can be modified independent of how the inputs are shaped. But how much you will unroll the RNN depends on your input length.

If i have a time series of 1 hour consisting of 1000 values, should i choose to unroll the RNN 1000 times, or is it too much?
My goal is to classify the whole time series, and ideally i want all 1000 time steps to be taken into account for this classification.


texts: Few activation functions handling various problems - neural networks

texts:
How can a few activation functions in neural networks handle so many different problems?

I know some basics theory behind ANN, but I can't get what functions like the sigmoid function etc. have in common with for example image classification?


texts: do the results change if you remove duplicate rows and you sum their weights?

texts:
Assuming that we have a dataset with the respondents on each row (N respondents) and their respective characteristics as columns (C characteristics). Each respondent has also a weight.

In case of high number of respondents, is it a good idea to remove the duplicate respondents and sum their weights ? Will this lead to different results ?

So my initial data would look like this

> dt
   id weight v1 v2
1:  1     10  2  4
2:  2     11  2  4
3:  3     12  2  4
4:  4     13  3  5
5:  5     14  3  5
6:  6     15  3  5

And since respondents 1,2,3 are the same, and respondents 4,5,6 are the same i would end up with this

> dt
   id weight v1 v2
1:  1     33  2  4
2:  2     42  3  5


texts: How to scrape a website with a searchbar

texts:
How do I scrape a website that basically looks like google with just a giant searchbar in the middle of the screen. From it you can search after various companies and their stats.

I have a list of 1000 companies I want to get information about. I want some bot to search each company from my list in the search bar, open the specific company's info window and extract a certain company code that exist on each page for each company.

Is there any easy and (of course) legal way to do it?


texts: Causation vs correlation problem in case of fixing other parameters and changing only one

texts:
I have a game that players play. D7 retention is defined as percentage of players who installed my game and after exactly 7 days later they played my game.

Now I am doing analytics and it shows that only 10% of players play my game on day 7. I change the difficulty of the game significantly and collect analytical data. Data shows that now D7 is 15%.

Can I conclude that D7 change is caused by difficulty change or this is one of the examples that correlation is not causation? And how I can identify if I can think of causation or not.

Physicists, psychologists in experiments have done a lot such things. Fixing all parameters changing one and observing the behavior change and drawing conclusion. What is the right way to approach this?


texts:
Let $\mathcal{S}$ be the training input data set where each input $u^i \in \mathcal{S}$  has $d$ features.
I want to design a ANN so that the cost function below is minimized (the sum of
square of pairwise differences between model outputs)  and the given constraint is satisfied, where $w$ is ANN model parameter vector.

Question: what kind of ANN is suitable for this purpose?[


texts: What are the main types of NLP annotators?

texts: I'm new to the world of text mining and have been reading up on annotators at places like the UIMA website. I'm encountering many new terms like named entity recognition, tokenizer, lemmatizer, gazetteer, etc. Coming from a layman background, this is all very confusing so can anyone tell me or link to resources that can explain what the main categories of annotators are and what they do?


texts: From my understanding you can use the transition matrix to predict the probability of going from the last predicted hidden state(state t), to the t+1 hidden state. My confusion is how in code format do I go from the hidden state predicted at time t+1 to the predicted observed state at time t+1.


texts: A statistic for testing if $\mu$ which is known to be in subspace $H$, is also in subspace $H_0\subseteq H$

texts:
I have a problem called problem 2.19 from Mathematische statistiek by A. van der Vaart https://staff.fnwi.uva.nl/b.j.k.kleijn/AsympStat-LecNotes2010.pdf

Before I state it, $P$ means the orthogonal projection onto $H$, $P_0$ same onto $H_0$ and $I-P$ same onto $H^{\bot}$. And $H_0$, $H$ are subspaces with $H_0\subseteq H\subseteq \mathbb{R}^n$. Now the problem is:

"Suppose that $Y\sim N_n(\mu,{\sigma}^2I)$, where $\mu$ and $\sigma^2$ are parameters that are known to belong a linear subspace $H$ and $(0,\infty)$, respectively. We wish to test $H_0: \mu\in H_0$ for a linear subspace $H_0\subseteq H$. As a test statistic we use $T=||P_0Y-PY||^2/||(I-P)Y||^2$. Show how an appropriate critical value can be obtained from a table of an $F$-distribution."

There is a theorem saying if $X\sim N(0,\Sigma)$ then $|X|^2\sim \sum_i \lambda_i Z_i^2$ with $\lambda_i$ all the eigenvalues of $\Sigma$ and $Z_i\sim N(0,1)$. By looking at the proof it seems like it's also possible to make a version for general $\mu$.

I know that we can make a basis of $H_0$ and by extending it in a right way we can get a basis of $H$. I also have an ugly calculation to get the distribution of $|P_0Y-PY|^2$ and $|(I-P)Y|^2$.

But I don't see how this statistic works. Does anyone have a comment or answer? Thanks in advance (it's written that I'd better not thank in the comments).


texts: How to classify/cluster clients of an application of transport like the Uber Model?

texts:
I have a question about User segmentation
I have a data-set of rides of an application that works at the same model as Uber.
The attributes I have : Reservation it means the id of the Ride and statutCourse means the status of the ride cancelled or finished .....
the Id client and the entreprise is the company where the client works IDChaufeur is conductor ID and i have the geographic coordinations of the start point and the finish point i also have date and hour of the ride also real time and estimated time and estimated distance

I want to classify/cluster clients profiles and conductors profiles

My problem is that one client can make many rides and I don't know if I can use many lines of rides for the same client in one dataset and than use this dataset to classify the clients into classes or profiles ?


texts: Maximum Likelihood with Gradient Descent or Coordinate Descent blows up

texts:
Context
The maximum likelihood estimators for a Normal distribution with unknown mean and unknown variance are
$$
\widehat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i \qquad \text{and} \qquad \widehat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n (x_i - \mu)^2
$$
These can be found (for example) by taking derivatives of the average log-likelihood
$$
\frac{1}{n}\sum_{i=1}^n \log p(x_i) = -\frac{1}{2}\log(2\pi) - \frac{1}{2n\sigma^2}\sum^n_{i=1} (x^{(i)} - \mu)^2 - \log \sigma
$$
Question: What if I want to use a gradient-based method?
Yes, I know I can just use the estimators found above. However, I want to find such estimators using a gradient-based method such as coordinate descent or gradient descent. These are the gradients with respect to $\mu$ and with respect to $\sigma$ (which you can set equal to zero to find the estimators above)
$$
\begin{align}
\frac{\partial}{\partial \mu} \frac{1}{n} \sum^n_{i=1} \log p(x^{(i)})
&= \frac{\overline{x}}{\sigma^2} - \frac{\mu}{\sigma^2} \\
\frac{\partial}{\partial \sigma} \frac{1}{n}\sum^n_{i=1} \log p(x^{(i)}) &= \frac{1}{n\sigma^3}\sum^n_{i=1}(x^{(i)} - \mu)^2 - \frac{1}{\sigma}
\end{align}
$$
I tried using them in gradient descent
$$
\begin{align}
\mu_{t+1} &\longleftarrow \mu_t + \gamma \left(\frac{\overline{x}}{\sigma^2_t} - \frac{\mu_t}{\sigma^2_t}\right) \\
\sigma_{t+1} &\longleftarrow \sigma_t + \gamma\left(\frac{1}{n\sigma^3_t}\sum^n_{i=1}(x^{(i)} - \mu_{t+1})^2 - \frac{1}{\sigma_t}\right)
\end{align}
$$
or in coordinate ascent (where I would keep, say $\sigma_t$ fixed and optimize $\mu_t$ for $n_{\text{inner}}$ times and then switch: keep $\mu_t$ fixed and optimize $\sigma_t$ for $n_{\text{inner}}$ times. All this for $n_{\text{outer}}$ times. However it seems to blow up for some reason and not give me the obvious answer. You can run the code here.

What am I doing wrong?


texts:
I have clustered vectors by cosine distance using nltk clusterer. If I understand correctly, Y axis for elbow method in euclidian distance would be the sum of every distance (squared) between centroid of the cluster with vectors that belongs to that cluster.

My question is: Would it be the same for clusters using cosine distance?

EDIT: ok, so i tried sum of squares with cosine distance and it seems, that it's returning the same values... heres my code:

EDIT2: My bad,is is working

from nltk.cluster import KMeansClusterer, cosine_distance
import numpy as np

#Load dataset obtained from http://cs.joensuu.fi/sipu/datasets/a1.txt
testing_vectors = np.loadtxt("a1.txt")

for k in range(1,10):
    kclusterer = KMeansClusterer(k, distance=cosine_distance)
    assigned_clusters = kclusterer.cluster(testing_vectors, assign_clusters=True)

    sum_of_squares = 0
    current_cluster = 0
    for centroid in kclusterer.means():
        current_page = 0
        for index_of_cluster_of_page in assigned_clusters:
            if index_of_cluster_of_page == current_cluster:
                y = testing_vectors[current_page]
                #sum_of_squares += np.sum((centroid - y) ** 2)
                sum_of_squares += (np.dot(centroid,y)**2)/(np.dot(centroid,centroid) * np.dot(y,y))
            current_page += 1
        current_cluster += 1

    print("for k=%s the sum of squares is:%s" %(k,sum_of_squares))
```


texts: Algorithm/Analysis that utilises incremental information?

texts:
I am looking for techniques which utilise information in an incremental manner.

Example: A day with inclement climate is likely to be followed by another day with inclement climate. Or when an entire dataset sorted by date is available, an algorithm able to identify that a person is likely to call in sick when the previous day had bad weather.

Is there any effective analysis which utilises this prior information where not all events are completely independent?